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ABSTRACT 

Change Group Analysis is suggested as a technique 
for examining the correlates of the differential effects of a 
compensatory education program within the group of enrolled pupils. 

It is based on using pretest-posttest relative changes for forming 
criterion groups and then using multiple dtscirainant analysis to 
uncover terns of predictor variables among relevant demographic, 
cognitive, and affective measures. The technique's potential for 
avoiding reliance on the artifactual properties of both regression 
phenomena and gain scores is discussed, it is suggested that the 
technique would be useful when post hoc matching or covariance 
analysis might produce misleading or even erroneous conclusions about 
the overall effectiveness of an intervention program. A confirmatory 
empirical study is proposed. (Author) 
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A technique for examining the correlates of the 
differential effects of a compensatory education program 
within the group of enrolled pupils Is suggested. It Is 
based on using pretest-posttest relative changes for forming 
"criterion” groups and then using multiple discriminant 
analysis to uncover patterns of "predictor” variables among 
relevant demographic, cognitive, and affective measures. 

Tne technique's potential for avoiding reliance on the 
artlfactual properties of both regression phenomena and gain 
scores Is discussed. It is suggested that the technique 
■would be useful when post hoc matching or covariance 
analysis might produce misleading or even erroneous 
conclusions about the overall effectiveness of an 
Intervention program. A confirmatory empirical study is 
proposed. 

Problem Statement 

Analyses of the benefits of compensatory education 
programs have usually been directed toward describing the 
overall effect of these programs on the entire groups of 
enrolled pupils. That Is, the post-treatment performance of 
these pupils on some criterion measure Is compared with 
their pre-treatment performance on that same measure or with 
the performance of some contrast group. Indeed, In some of 
the more elegant designs, both pre-post absolute changes and 
the performance of some constrast group might be employed. 
These evaluation strategies re ertlclngly simple and 
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straightforward, but they potentially can lead to biased, 
and even Incorrect, conclusions. This point will be 
elaborated upon later in this paper# 

At the moment, however, it is more appropriate to 
proceed with the main purpose of this paper, which is to 
suggest an alternate evaluation strategy# This is a 
technique for discovering the coi-relates of relatively 
different success patterns within the group of pupils 
enrolled in the program# The assumptions are thut (a) not 
all of the pupils will have benefited equally from the 
treatment and (b) there are meaningful and systematic 
correlates of these differential success patterns. The 
proposed technique is addressed to such questions as, "What 
kinds of pupils, from which sorts of homes, receiving what 
types of instruction in which sorts of programs, and in what 
kinds of communities, do relatively well (or poorly) In 
Program Such-and-Such?" 

Kethod 

This proposed teohnlque, "Change Group Analysis" (CGA), 
Is a combination of a simple ranking procedure and a 
powerful multivariate analytio method. Pupils are ranked on 
the basis of their performance on one of the customary pre- 
and post-treatment measures, such as an achievement test.- 
Then the distributions of these measuros are divided Into 
equal numbers of pupils# For expository purposes, they are 
here divided into thirds, as in Figure 1# In praotioe, the 
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number of splits to be made would depend on both the number 
of pupils Involved and the strength and shape of the 
relationship between the marginal variables* The bivariate 
distribution* when the marginals are split Into thirds* Is 
thus divided Into up to nine "Change Groups" (CGs). The 
designations* "low-low" (L-L), "low-middle" (L-M), and so 
forth, lndloato the Initial and 'Inal relative positions of 
the members of eaoh GG. 

Next* all of the' available and possibly relevant data 
on the pupils are used as "predictor" variables for multiple 
dlsorlmlnant analysis (Cooley & Lohnes, 1962j Rae, 1952 1 
Tatsuoka & Tledeman, 195^)» seeking maximum discrimination 
among the CGs, Potential predlotor variables Include the 
wealth of demographlo data typloally gathered In compensatory 
education evaluations those on the pupils themselves* their 
families* teaohers* sohools* and communities. These dr.ta 
Also lnolude the cognitive and affeotive measures obtained 
In the oourse of the Intervention program. Indeed, the 
variables used for forming the CGs for one CGA are potential 
predlotor variables for other CGAs. 

Discussion 

The dlsousslon of CGA must begin with what It Is not. 
That Is because* at first glanoe* CGA might appear to be a 
devloe for capitalizing on regression effeots or for 
performing an analysis on gain soores without taking Into 

aooount the methodological problems associated with gain 
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scores . 

Whon two tosts are Imperfectly correlated, oxamlnoes 
below the mean on one test are predicted, through the use of 
a linear regression equation to be below the mean on the 
other test, but not as far below, In standard deviation 
units, This predicted statistical "regression toward the 
mean" may or may not actually occur in faot (Hays & Winkler, 
1970). When it does, we terra It a "regression effect," 

When persons are selected for an experiment (or for an 
intervention program) beoause they deviate from the mean on 
some measure or on a variable related to that measure, and 
when that same measure is used as a posttest, these 
regression effects may then provide souroes of invalidity 
for the interpretation of the results of the experiment 
(Campbell & Stanley, 1963)* 

But in CGA, it is the pupils' relative scores, rather 
than their absolute scores, that are used as the basis of 
the pupils' C G assignments. Neither a systematic movement 
of the group as a whole toward its population mean nor an 
arti factual oonstrlotion of the posttest variance would 
, affect these relative standings. 

On the other hand, suppose that the pupils in the 
program were members of different populations, with 
population parameters. If regression ef foots were then 
operating, the members of the various populations would be 
regressing toward different moans and would be. doing so at 
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different rates. This Introduces a dilemma. Considering 
all of the participating pupils to be members of the same 
population, one in which there are meaningful subgroups 
whose combinations of personal and program demographic, 
cognitive, and affective attributes lead to different 
relative changes, Is the purpose of CGA. Even if all or 
part of these patterns of change are due to different 
regression effects, the fact Is that those changes still 
will have taken plaoe*- This Is the dilemmai are these 
differential change patterns the result of the varying 
sensitivities of different subgroups to the treatment 
manifolds or Is It merely that a complexity of different 
regression effeots Is being observed? It my very well be 
that the extent to whioh program- and teacher-related 
variables relate to CG membership will determine which 
hypothesis will be accepted. 

Although the regression-effects question is one of 
obvious complexity, that surrounding the utilization of 
change scores is simpler. The methodological problems 
associated with the interpretation of change scores is well 
documented. See, for instance, Harris (1964), But, since 
CGA relies on changes in relative, rather than absolute, 
performance, it is not, stziotly speaking, based on ohange 
scores. If it were, and if it were the bivariate 
distribution, not the marginals, that were. divided, the CGs 
would be formed as in Figure 2*. Butlwith the CGs formed as 
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FIGURE 2. GROUPS FORMED ON THE BASIS OF GAIN SCORES 
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in Figure 1, L-L, K-M, and H-H are considered to be separate 
groups, although the members will have had approximately 
equal chango scores (if the pretest and posttest scores are 
related in a linear fashion) • This is also obviously true 
for those in L-K and H-H and also for those in M-L and H-H, 
Uith multiple discriminant analysis, the differences among 
all of the CGs are maximized. That is, the intent is to 
separate L-L and K-M, which will have had somewhat similar 
performances, as much as L-H and H-L, whose performances 
will have differed greatly. 

As already mentioned, tho use of relative standings 
obviates the assumption of linearity of regression of 
posttest on pretest. It should also be noted that these 
measures need not be assumed to have attained on interval 
scale. 

At the outset of this paper, it was suggested that CGA 
be employed for examining the relative differential effects 
of a compensatory program within tho group of enrolled 
pupils, somewhat as a supplement to the more traditional 
approaches which focus on the absolute performances of those 
in the group as a whole, usually comparing the group 
performance with that of some contrast group. Often either 
this contrast group is selected through a post hoc matching 
technique or else the data analysis is performed with 
covariance analysis. For some time, the problems of 

interpretation following p ost hoc matohing have been well 
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known (Campbell & Stanley, 1963)* But covariance analysis 
can also lead to misleading and even erroneous conclusions 
(Campbell & Erlebacher, In press; Lord 1967 • 1969)* The 
point here Is that CGA might provide the only suitable 
alternative for evaluation of a program when the evaluator 
is faced with a necessary but completely ex post facto 
Investigation, 

Further Research 

The next step Is to perform CGA on some real data. 
Artificially-generated data would be unsatisfactory because 
the rules for generating these data would have to be too 
deterministic. In any event, a very large number of pupils, 
on each of whom a great many relevant measures are 
available, would be needed. This would normally be the case 
with CGA, It Is Intended for the evaluation of projeots 
that are large-scale, at least regional, If not national, In 
scope. For a confirmatory study, an even greater-than-usual 
r mber of pupils will be needea. The reason Is that this 
group should be randomly divided such that two or more 
parallel CGAs could be performed. The agr t »ment of the 
results obtained therefrom would provide svlder.oe for the 
consistency of the technique. 
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